Date: Tue, 2 Aug 2022 04:32:31 +0200 From: "CIKM 2022 Short Paper Track" To: Hossein Fani Subject: CIKM2022 Short Paper Track notification for paper 0781 Sender: cikm2022-short@easychair.org Return-Path: cikm2022-short@easychair.org Dear Hossein Fani I am very pleased to inform you that your paper Research: Effective Neural Team Formation via Negative Samples has been accepted for presentation at CIKM’22 Short Paper track. Attached below are the reviewer's comments, which we hope you will find helpful for your preparation of the camera-ready version. Instructions for producing the camera-ready copy will be sent to you soon. The camera-ready deadline is *** August 15, 2022 ***. If you have any question please let us know. Thank you for submitting to CIKM’22, and please accept our congratulations on your paper's acceptance. Sincerely, Wendy Wang and Latifur Khan SUBMISSION: 0781 TITLE: Research: Effective Neural Team Formation via Negative Samples ------------------------- METAREVIEW ------------------------ All reviewers agree that the idea and contribution of this paper are interesting and solid. While there are some concerns about the method and experiment details, it is clear that this paper is above the bar for acceptance. For camera ready, I suggest the authors to carefully read the reviewers' comments and revise the paper accordingly. ----------------------- REVIEW 1 --------------------- SUBMISSION: 0781 TITLE: Research: Effective Neural Team Formation via Negative Samples AUTHORS: Arman Dashti, Saeed Samet and Hossein Fani ----------- Overall evaluation ----------- SCORE: 1 (weak accept) ----- TEXT: This paper proposes an optimization objective that leverages negative samples in addition to positive ones. The idea is of course appealing, although the real challenge is to get the negative samples for training in the application domain considered here, i.e., collaborative team formation. Overall this seems to be a decent contribution. But for the conclusions reached, I would have preferred at least 2 more datasets to be experimented with one with a similar distribution of DBLP and the other with a similar distribution of IMDB. ----------- Strengths and reasons to accept ----------- The idea is appealing. The research questions formulated and handled are interesting and meaningful. The paper is quite well-written. ----------- Weaknesses and limitations ----------- The conclusions reached may be premature. More datasets should be investigated. ----------------------- REVIEW 2 --------------------- SUBMISSION: 0781 TITLE: Research: Effective Neural Team Formation via Negative Samples AUTHORS: Arman Dashti, Saeed Samet and Hossein Fani ----------- Overall evaluation ----------- SCORE: 1 (weak accept) ----- TEXT: This paper addresses the task of forming successful teams based on the pool of available candidates and their skill set. The main contribution is the novel idea of leveraging unsuccessful teams, which is technically implemented as negative sampling. The experiments show that negative sampling generally improves performance, but the exact impact varies according to dataset and the underlying model. ----------- Strengths and reasons to accept ----------- - The idea of introducing negative sampling is intuitive and sound. - The paper is well written and easy to follow. I particularly like how the research questions and the insights are well organized and visually easy to spot. ----------- Weaknesses and limitations ----------- Weaknesses: - The authors present the problem under study as a well-known and thus well-studied phenomenon. However, only one prior work (reference [14] in the paper) was used a baseline. I am not an expert in this field, but imagine there must be other prior work to compare with. - The results show that negative sampling, at least in its currently presented form, are sensitive to dataset and the underlying model. In other words, the is room for improvement for the negative sampling strategies. Typographical issues: - Figure 1: Are the y-axes of #members correct? Are there actually teams of > 10^5 members?! - Tables 2 and 3: The heat map-style shading needs to be explicitly explained. Particularly important is if the comparison is made against random or the "plain" version of the respective models. - "(4) Bayesian neural models that benefit from negative samples outperforms other models in less number of training epochs..." Training epochs are countable, thus "fewer" and not "less." ----------------------- REVIEW 3 --------------------- SUBMISSION: 0781 TITLE: Research: Effective Neural Team Formation via Negative Samples AUTHORS: Arman Dashti, Saeed Samet and Hossein Fani ----------- Overall evaluation ----------- SCORE: 2 (accept) ----- TEXT: The authors proposed three negative sampling huheuristics, which help the forming team task leverage the unsucessful teams when using the neural models. To demonstrate the effectiveness of the proposed methods, the authors intergrate their negative sampling methods to feed-forward non-bayesian and bayesian neural networks. The experiments show that negative sampling improved the effectiveness of Bayesian neural models for the team forming task. The authors also show other usefull findings. I believe the negative idea and the experiments result can provide good contribution for this task and our whole community. ----------- Strengths and reasons to accept ----------- 1. Neat idea but good results. The experiments design are also very good. 2. The experiments results not only demonstrate the effectiness of the idea, but also provide some empirical results for future researchers in this field. 3. This paper is well orginized and easily reading. ----------- Weaknesses and limitations ----------- - The authors should give more detail like how many "experts" and unsucessful "experts" in each dataset (imdb and dblp). Although the figure 1 contains some infromation, it is not enough. They should put the detail in the 4.1.1 section.